Managing configuration and sensitive data for workloads in a virtualized computing system

ABSTRACT

An example virtualized computing system includes: a host cluster having a virtualization layer directly executing on hardware platforms of hosts, the virtualization layer supporting execution of virtual machines (VMs), the VMs including pod VMs and native VMs, the pod VMs including container engines supporting execution of containers in the pod VMs, the native VMs including applications executing on guest operating systems; an orchestration control plane integrated with the virtualization layer and including a master server and native VM controllers, the master server managing lifecycles of the pod VMs and the native VMs; and management agents, executing in the native VMs, configured to receive decoupled information from the master server through the native VM controllers and to provide the decoupled information for consumption by the applications executing in the native VMs, the decoupled information including at least one of configuration information and secret information.

Applications today are deployed onto a combination of virtual machines(VMs), containers, application services, and more. For deploying suchapplications, a container orchestrator (CO) known as Kubernetes® hasgained in popularity among application developers. Kubernetes provides aplatform for automating deployment, scaling, and operations ofapplication containers across clusters of hosts. It offers flexibilityin application development and offers several useful tools for scaling.

In a Kubernetes system, containers are grouped into logical unit called“pods” that execute on nodes in a cluster (also referred to as “nodecluster”). Containers in the same pod share the same resources andnetwork and maintain a degree of isolation from containers in otherpods. The pods are distributed across nodes of the cluster. In a typicaldeployment, a node includes an operating system (OS), such as Linux®,and a container engine executing on top of the OS that supports thecontainers of the pod. A node can be a physical server or a VM.

Modern workload management software such as Kubernetes allow you todecouple configuration and sensitive information from pods and make theapplications mode secure and portable. This separation also allows formanaging and updating the configuration and sensitive informationwithout updating application code, allowing for live updates of thisinformation. The configuration information can include, for example,configuration files, environment variables, and the like. Sensitiveinformation can include, for example, passwords, secure shell (SSH)keys, tokens, and the like. Workload management software allows forpassing this information to the pods in various ways, including settingenvironment variables, creating files containing this information, andthe like. It is desirable to extend these features of the workloadmanagement software for use with applications executing outside of a podcontext, such as within a VM.

SUMMARY

In an embodiment, a virtualized computing system includes: a hostcluster having a virtualization layer directly executing on hardwareplatforms of hosts, the virtualization layer supporting execution ofvirtual machines (VMs), the VMs including pod VMs and native VMs, thepod VMs including container engines supporting execution of containersin the pod VMs, the native VMs including applications executing on guestoperating systems; an orchestration control plane integrated with thevirtualization layer, the orchestration control plane including a masterserver and native VM controllers, the master server managing lifecyclesof the pod VMs and the native VMs, the native VM controllers executingin the virtualization layer external to the VMs and configured as agentsof the master server to manage the VMs; and management agents, executingin the native VMs, configured to receive decoupled information from themaster server through the native VM controllers and to provide thedecoupled information for consumption by the applications executing inthe native VMs, the decoupled information including at least one ofconfiguration information and secret information.

In another embodiment, a host computer in a host cluster of avirtualized computing system includes: a hardware platform; avirtualization layer, directly executing on the hardware platform,supporting execution of virtual machines (VMs), the VMs including podVMs and native VMs, the pod VMs including container engines supportingexecution of containers in the pod VMs, the native VMs includingapplications executing on guest operating systems; a native VMcontroller, executing in the virtualization layer external to the VMs,configured as an agent of an orchestration control plane of thevirtualized computing system, the native VM controller configured tomanage the native VMs; and management agents, executing in the nativeVMs, configured to receive decoupled information from the native VMcontrollers and to provide the decoupled information for consumption bythe applications executing in the native VMs, the decoupled informationincluding at least one of configuration information and secretinformation managed by the orchestration control plane.

In another embodiment, a method of application orchestration in avirtualized computing system is described. The virtualized computingsystem includes a host cluster having a virtualization layer directlyexecuting on hardware platforms of hosts, the virtualization layersupporting execution of virtual machines (VMs), the virtualization layerintegrated with an orchestration control plane. The method includes:receiving, at a master server of the orchestration control plane,specification data for an application; deploying, based on thespecification data, the application to a native VM the VMs, the nativeVM executing on the virtualization layer, the native VM executing amanagement agent configured as an agent of a VM controller in thevirtualization layer; receiving decoupled information at the managementagent from a master server of the orchestration control plane throughthe native VM controller; and providing the decoupled information forconsumption by the applications executing in the native VMs, thedecoupled information including at least one of configurationinformation and secret information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a virtualized computing system in whichembodiments described herein may be implemented.

FIG. 2 is a block diagram depicting a software platform according anembodiment.

FIG. 3 is a block diagram of a supervisor Kubernetes master according toan embodiment.

FIG. 4 is a block diagram depicting a logical view of a guest clusterexecuting in a virtualized computing system according to an embodiment.

FIG. 5 is a flow diagram depicting a method of application orchestrationin a virtualized computing system according to an embodiment.

DETAILED DESCRIPTION

Techniques for managing configuration and secret information forapplications in a virtualized computing system are described. Inembodiments described herein, the virtualized computing system includesa cluster of physical servers (“hosts”) referred to as a “host cluster.”The host cluster includes a virtualization layer, executing on hardwareplatforms of the hosts, which supports execution of virtual machines(VMs). A virtualization management server manages the host cluster, thevirtualization layer, and the VMs executing thereon. The virtualizedcomputing system includes shared storage accessible by the host cluster.The container orchestrator executes in the virtualized computing system(e.g., on one or more VMs) and is configured to deploy and manageapplications in the host cluster. In embodiments, the containerorchestrator is a Kubernetes system that deploys and managescontainerized applications in a cluster of VMs (a “Kubernetes cluster”).

In one or more embodiments, the orchestration control plane comprises asupervisor container orchestrator having extensions that cooperate withthe virtualization management server and agents installed in thevirtualization layer. A host cluster having the orchestration controlplane is referred to herein as a “supervisor cluster.” A user interactswith the orchestration control plane to deploy and manage applicationsexecuting on the supervisor cluster. In embodiments, the orchestrationcontrol plane uses hosts to implement nodes, and VMs to implement pods,of a Kubernetes cluster. Kubernetes pods are implemented as “pod VMs,”each of which includes a kernel and container engine that supportsexecution of containers. The container orchestrator (e.g., Kubernetes)executes in VMs alongside the pod VMs. The container orchestrator isfurther configured to implement applications in native VMs alongsidepods (e.g., non-containerized applications). Techniques are described toprovide and update configuration/secret information used by applicationsexecuting in native VMs. These and further advantages and aspects of thedisclosed techniques are described below with respect to the drawings.

FIG. 1 is a block diagram of a virtualized computing system 100 in whichembodiments described herein may be implemented. System 100 includes acluster of hosts 120 (“host cluster 118”) that may be constructed onserver-grade hardware platforms such as an x86 architecture platforms.For purposes of clarity, only one host cluster 118 is shown. However,virtualized computing system 100 can include many of such host clusters118. As shown, a hardware platform 122 of each host 120 includesconventional components of a computing device, such as one or morecentral processing units (CPUs) 160, system memory (e.g., random accessmemory (RAM) 162), one or more network interface controllers (NICs) 164,and optionally local storage 163. CPUs 160 are configured to executeinstructions, for example, executable instructions that perform one ormore operations described herein, which may be stored in RAM 162. NICs164 enable host 120 to communicate with other devices through a physicalnetwork 180. Physical network 180 enables communication between hosts120 and between other components and hosts 120 (other componentsdiscussed further herein). Physical network 180 can include a pluralityof VLANs to provide external network virtualization as described furtherherein.

In the embodiment illustrated in FIG. 1, hosts 120 access shared storage170 by using NICs 164 to connect to network 180. In another embodiment,each host 120 contains a host bus adapter (HBA) through whichinput/output operations (IOs) are sent to shared storage 170 over aseparate network (e.g., a fibre channel (FC) network). Shared storage170 include one or more storage arrays, such as a storage area network(SAN), network attached storage (NAS), or the like. Shared storage 170may comprise magnetic disks, solid-state disks, flash memory, and thelike as well as combinations thereof. In some embodiments, hosts 120include local storage 163 (e.g., hard disk drives, solid-state drives,etc.). Local storage 163 in each host 120 can be aggregated andprovisioned as part of a virtual SAN, which is another form of sharedstorage 170.

A software platform 124 of each host 120 provides a virtualizationlayer, referred to herein as a hypervisor 150, which directly executeson hardware platform 122. In an embodiment, there is no interveningsoftware, such as a host operating system (OS), between hypervisor 150and hardware platform 122. Thus, hypervisor 150 is a Type-1 hypervisor(also known as a “bare-metal” hypervisor). As a result, thevirtualization layer in host cluster 118 (collectively hypervisors 150)is a bare-metal virtualization layer executing directly on host hardwareplatforms. Hypervisor 150 abstracts processor, memory, storage, andnetwork resources of hardware platform 122 to provide a virtual machineexecution space within which multiple virtual machines (VM) may beconcurrently instantiated and executed. One example of hypervisor 150that may be configured and used in embodiments described herein is aVMware ESXi™ hypervisor provided as part of the VMware vSphere® solutionmade commercially available by VMware, Inc. of Palo Alto, Calif.

In the example of FIG. 1, host cluster 118 is enabled as a “supervisorcluster,” described further herein, and thus VMs executing on each host120 include pod VMs 130 and native VMs 140. A pod VM 130 is a virtualmachine that includes a kernel and container engine that supportsexecution of containers, as well as an agent (referred to as a pod VMagent) that cooperates with a controller of an orchestration controlplane 115 executing in hypervisor 150 (referred to as a pod VMcontroller). An example of pod VM 130 is described further below withrespect to FIG. 2. VMs 130/140 support applications 141 deployed ontohost cluster 118, which can include containerized applications (e.g.,executing in either pod VMs 130 or native VMs 140) and applicationsexecuting directly on guest operating systems (non-containerized)(e.g.,executing in native VMs 140). One specific application discussed furtherherein is a guest cluster executing as a virtual extension of asupervisor cluster. Some VMs 130/140, shown as support VMs 145, havespecific functions within host cluster 118. For example, support VMs 145can provide control plane functions, edge transport functions, and thelike. An embodiment of software platform 124 is discussed further belowwith respect to FIG. 2.

Host cluster 118 is configured with a software-defined (SD) networklayer 175. SD network layer 175 includes logical network servicesexecuting on virtualized infrastructure in host cluster 118. Thevirtualized infrastructure that supports the logical network servicesincludes hypervisor-based components, such as resource pools,distributed switches, distributed switch port groups and uplinks, etc.,as well as VM-based components, such as router control VMs, loadbalancer VMs, edge service VMs, etc. Logical network services includelogical switches, logical routers, logical firewalls, logical virtualprivate networks (VPNs), logical load balancers, and the like,implemented on top of the virtualized infrastructure. In embodiments,virtualized computing system 100 includes edge transport nodes 178 thatprovide an interface of host cluster 118 to an external network (e.g., acorporate network, the public Internet, etc.). Edge transport nodes 178can include a gateway between the internal logical networking of hostcluster 118 and the external network. Edge transport nodes 178 can bephysical servers or Is. For example, edge transport nodes 178 can beimplemented in support VMs 145 and include a gateway of SD network layer175. Various clients 119 can access service(s) in virtualized computingsystem through edge transport nodes 178 (including VM management client106 and Kubernetes client 102, which as logically shown as beingseparate by way of example).

Virtualization management server 116 is a physical or virtual serverthat manages host cluster 118 and the virtualization layer therein.Virtualization management server 116 installs agent(s) 152 in hypervisor150 to add a host 120 as a managed entity. Virtualization managementserver 116 logically groups hosts 120 into host cluster 118 to providecluster-level functions to hosts 120, such as N migration between hosts120 (e.g., for load balancing), distributed power management, dynamic VMplacement according to affinity and anti-affinity rules, andhigh-availability. The number of hosts 120 in host cluster 118 may beone or many. Virtualization management server 116 can manage more thanone host cluster 118.

In an embodiment, virtualization management server 116 further enableshost cluster 118 as a supervisor cluster 101. Virtualization managementserver 116 installs additional agents 152 in hypervisor 150 to add host120 to supervisor cluster 101. Supervisor cluster 101 integrates anorchestration control plane 115 with host cluster 118. In embodiments,orchestration control plane 115 includes software components thatsupport a container orchestrator, such as Kubernetes, to deploy andmanage applications on host cluster 118. By way of example, a Kubernetescontainer orchestrator is described herein. In supervisor cluster 101,hosts 120 become nodes of a Kubernetes cluster and pod VMs 130 executingon hosts 120 implement Kubernetes pods. Orchestration control plane 115includes supervisor Kubernetes master 104 and agents 152 executing invirtualization layer (e.g., hypervisors 150). Supervisor Kubernetesmaster 104 includes control plane components of Kubernetes, as well ascustom controllers, custom plugins, scheduler extender, and the likethat extend Kubernetes to interface with virtualization managementserver 116 and the virtualization layer. For purposes of clarity,supervisor Kubernetes master 104 is shown as a separate logical entity.For practical implementations, supervisor Kubernetes master 104 isimplemented as one or more VM(s) 130/140 in host cluster 118. Further,although only one supervisor Kubernetes master 104 is shown, supervisorcluster 101 can include more than one supervisor Kubernetes master 104in a logical cluster for redundancy and load balancing.

In an embodiment, virtualized computing system 100 further includes astorage service 110 that implements a storage provider in virtualizedcomputing system 100 for container orchestrators. In embodiments,storage service 110 manages lifecycles of storage volumes (e.g., virtualdisks) that back persistent volumes used by containerized applicationsexecuting in host cluster 118. A container orchestrator such asKubernetes cooperates with storage service 110 to provide persistentstorage for the deployed applications. In the embodiment of FIG. 1,supervisor Kubernetes master 104 cooperates with storage service 110 todeploy and manage persistent storage in the supervisor clusterenvironment. Other embodiments described below include a vanillacontainer orchestrator environment and a guest cluster environment.Storage service 110 can execute in virtualization management server 116as shown or operate independently from virtualization management server116 (e.g., as an independent physical or virtual server).

In an embodiment, virtualized computing system 100 further includes anetwork manager 112. Network manager 112 is a physical or virtual serverthat orchestrates SD network layer 175. In an embodiment, networkmanager 112 comprises one or more virtual servers deployed as VMs.Network manager 112 installs additional agents 152 in hypervisor 150 toadd a host 120 as a managed entity, referred to as a transport node. Inthis manner, host cluster 118 can be a cluster 103 of transport nodes.One example of an SD networking platform that can be configured and usedin embodiments described herein as network manager 112 and SD networklayer 175 is a VMware NSX® platform made commercially available byVMware, Inc. of Palo Alto, Calif.

Network manager 112 can deploy one or more transport zones invirtualized computing system 100, including VLAN transport zone(s) andan overlay transport zone. A VLAN transport zone spans a set of hosts120 (e.g., host cluster 118) and is backed by external networkvirtualization of physical network 180 (e.g., a VLAN). One example VLANtransport zone uses a management VLAN 182 on physical network 180 thatenables a management network connecting hosts 120 and the VI controlplane (e.g., virtualization management server 116 and network manager112). An overlay transport zone using overlay VLAN 184 on physicalnetwork 180 enables an overlay network that spans a set of hosts 120(e.g., host cluster 118) and provides internal network virtualizationusing software components (e.g., the virtualization layer and servicesexecuting in VMs). Host-to-host traffic for the overlay transport zoneis carried by physical network 180 on the overlay VLAN 184 usinglayer-2-over-layer-3 tunnels. Network manager 112 can configure SDnetwork layer 175 to provide a cluster network 186 using the overlaynetwork. The overlay transport zone can be extended into at least one ofedge transport nodes 178 to provide ingress/egress between clusternetwork 186 and an external network.

In an embodiment, system 100 further includes an image registry 190. Asdescribed herein, containers of supervisor cluster 101 execute in podVMs 130. The containers in pod VMs 130 are spun up from container imagesmanaged by image registry 190. Image registry 190 manages images andimage repositories for use in supplying images for containerizedapplications.

Virtualization management server 116 and network manager 112 comprise avirtual infrastructure (VI) control plane 113 of virtualized computingsystem 100. Virtualization management server 116 can include asupervisor cluster service 109, storage service 110, and VI services108. Supervisor cluster service 109 enables host cluster 118 assupervisor cluster 101 and deploys the components of orchestrationcontrol plane 115. VI services 108 include various virtualizationmanagement services, such as a distributed resource scheduler (DRS),high-availability (HA) service, single sign-on (SSO) service,virtualization management daemon, and the like. DRS is configured toaggregate the resources of host cluster 118 to provide resource poolsand enforce resource allocation policies. DRS also provides resourcemanagement in the form of load balancing, power management, VMplacement, and the like. HA service is configured to pool VMs and hostsinto a monitored cluster and, in the event of a failure, restart VMs onalternate hosts in the cluster. A single host is elected as a master,which communicates with the HA service and monitors the state ofprotected VMs on subordinate hosts. The HA service uses admissioncontrol to ensure enough resources are reserved in the cluster for VMrecovery when a host fails. SSO service comprises security tokenservice, administration server, directory service, identity managementservice, and the like configured to implement an SSO platform forauthenticating users. The virtualization management daemon is configuredto manage objects, such as data centers, clusters, hosts, VMs, resourcepools, datastores, and the like.

A VI admin can interact with virtualization management server 116through a VM management client 106. Through VM management client 106, aVI admin commands virtualization management server 116 to form hostcluster 118, configure resource pools, resource allocation policies, andother cluster-level functions, configure storage and networking, enablesupervisor cluster 101, deploy and manage image registry 190, and thelike.

Kubernetes client 102 represents an input interface for a user tosupervisor Kubernetes master 104. Kubernetes client 102 is commonlyreferred to as kubectl. Through Kubernetes client 102, a user submitsdesired states of the Kubernetes system, e.g., as YAML documents, tosupervisor Kubernetes master 104. In embodiments, the user submits thedesired states within the scope of a supervisor namespace. A “supervisornamespace” is a shared abstraction between VI control plane 113 andorchestration control plane 115. Each supervisor namespace providesresource-constrained and authorization-constrained units ofmulti-tenancy. A supervisor namespace provides resource constraints,user-access constraints, and policies (e.g., storage policies, networkpolicies, etc.). Resource constraints can be expressed as quotas,limits, and the like with respect to compute (CPU and memory), storage,and networking of the virtualized infrastructure (host cluster 118,shared storage 170, SD network layer 175). User-access constraintsinclude definitions of users, roles, permissions, bindings of roles tousers, and the like. Each supervisor namespace is expressed withinorchestration control plane 115 using a namespace native toorchestration control plane 115 (e.g., a Kubernetes namespace orgenerally a “native namespace”), which allows users to deployapplications in supervisor cluster 101 within the scope of supervisornamespaces. In this manner, the user interacts with supervisorKubernetes master 104 to deploy applications in supervisor cluster 101within defined supervisor namespaces.

While FIG. 1 shows an example of a supervisor cluster 101, thetechniques described herein do not require a supervisor cluster 101. Insome embodiments, host cluster 118 is not enabled as a supervisorcluster 101. In such case, supervisor Kubernetes master 104, Kubernetesclient 102, pod VMs 130, supervisor cluster service 109, and imageregistry 190 can be omitted. While host cluster 118 is show as beingenabled as a transport node cluster 103, in other embodiments networkmanager 112 can be omitted. In such case, virtualization managementserver 116 functions to configure SD network layer 175.

FIG. 2 is a block diagram depicting software platform 124 according anembodiment. As described above, software platform 124 of host 120includes hypervisor 150 that supports execution of VMs, such as pod VMs130, native VMs 140, and support VMs 145. In an embodiment, hypervisor150 includes a VM management daemon 213, a host daemon 214, a pod VMcontroller 216, a native VM controller 217, an image service 218, andnetwork agents 222. VM management daemon 213 is an agent 152 installedby virtualization management server 116. VM management daemon 213provides an interface to host daemon 214 for virtualization managementserver 116. Host daemon 214 is configured to create, configure, andremove VMs (e.g., pod VMs 130 and native VMs 140).

Pod VM controller 216 is an agent 152 of orchestration control plane 115for supervisor cluster 101 and allows supervisor Kubernetes master 104to interact with hypervisor 150. Pod VM controller 216 configures therespective host as a node in supervisor cluster 101. Pod VM controller216 manages the lifecycle of pod VMs 130, such as determining when tospin-up or delete a pod VM. Pod VM controller 216 also ensures that anypod dependencies, such as container images, networks, and volumes areavailable and correctly configured. Pod VM controller 216 is omitted ifhost cluster 118 is not enabled as a supervisor cluster 101. Native VMcontroller is an agent 152 of orchestration control plane 115 forsupervisor cluster 101 and allows supervisor Kubernetes master 104 tointeract with hypervisor 150 to manage lifecycles of native VMs 140 andapplications executing therein. While shown separately from pod VMcontroller 216, in some embodiments both pod VM controller 216 andnative VM controller 217 can be functions of a single controller.

Image service 218 is configured to pull container images from imageregistry 190 and store them in shared storage 170 such that thecontainer images can be mounted by pod VMs 130. Image service 218 isalso responsible for managing the storage available for container imageswithin shared storage 170. This includes managing authentication withimage registry 190, assuring providence of container images by verifyingsignatures, updating container images when necessary, and garbagecollecting unused container images. Image service 218 communicates withpod VM controller 216 during spin-up and configuration of pod VMs 130.In some embodiments, image service 218 is part of pod VM controller 216.In embodiments, image service 218 utilizes system VMs 130/140 in supportVMs 145 to fetch images, convert images to container image virtualdisks, and cache container image virtual disks in shared storage 170.

Network agents 222 comprises agents 152 installed by network manager112. Network agents 222 are configured to cooperate with network manager112 to implement logical network services. Network agents 222 configurethe respective host as a transport node in a cluster 103 of transportnodes.

Each pod VM 130 has one or more containers 206 running therein in anexecution space managed by container engine 208. The lifecycle ofcontainers 206 is managed by pod VM agent 212. Both container engine 208and pod VM agent 212 execute on top of a kernel 210 (e.g., a Linux®kernel). Each native VM 140 has applications 202 running therein on topof an OS 204. Native VMs 140 do not include pod VM agents and areisolated from pod VM controller 216. Rather, native VMs 140 includemanagement agents 213 that communicate with native VM controller 217.Container engine 208 can be an industry-standard container engine, suchas libcontainer, runc, or containerd. Pod VMs 130, pod VM controller216, native VM controller 217, and image service 218 are omitted if hostcluster 118 is not enabled as a supervisor cluster 101.

FIG. 3 is a block diagram of supervisor Kubernetes master 104 accordingto an embodiment. Supervisor Kubernetes master 104 includes applicationprogramming interface (API) server 302, a state database 303, ascheduler 304, a scheduler extender 306, controllers 308, and plugins319. API server 302 includes the Kubernetes API server, kube-api-server(“Kubernetes API 326”) and custom APIs 305. Custom APIs 305 are APIextensions of Kubernetes API 326 using either the customresource/operator extension pattern or the API extension server pattern.Custom APIs 305 are used to create and manage custom resources, such asVM objects. API server 302 provides a declarative schema for creating,updating, deleting, and viewing objects.

State database 303 stores the state of supervisor cluster 101 (e.g.,etcd) as objects created by API server 302. A user can provideapplication specification data to API server 302 that defines variousobjects supported by the API (e.g., as a YAML document). The objectshave specifications that represent the desired state. State database 303stores the objects defined by application specification data as part ofthe supervisor cluster state. Standard Kubernetes objects (“Kubernetesobjects 310”) include namespaces 320, nodes 322, pods 324, config maps215, secrets 327, among others. Custom objects are resources definedthrough custom APIs 305 (e.g., VM objects 307). Namespaces 320 providescope for objects. Namespaces are objects themselves maintained in statedatabase 303. A namespace can include resource quotas, limit ranges,role bindings, and the like that are applied to objects declared withinits scope. VI control plane 113 creates and manages supervisornamespaces for supervisor cluster 101. A supervisor namespace is aresource-constrained and authorization-constrained unit of multi-tenancymanaged by virtualization management server 116. Namespaces 320 inheritconstraints from corresponding supervisor cluster namespaces. Configmaps 325 include configuration information for applications managed bysupervisor Kubernetes master 104. Secrets 327 include sensitiveinformation for use by applications managed by supervisor Kubernetesmaster 104 (e.g., passwords, keys, tokens, etc.). The configurationinformation and the secret information stored by config maps 325 andsecrets 327 is generally referred to herein as decoupled information.Decoupled information is information needed by the managed applications,but which is decoupled from the application code.

Controllers 308 can include, for example, standard Kubernetescontrollers (“Kubernetes controllers 316”) (e.g.,kube-controller-manager controllers, cloud-controller-managercontrollers, etc.) and custom controllers 318. Custom controllers 318include controllers for managing lifecycle of Kubernetes objects 310 andcustom objects. For example, custom controllers 318 can include a VMcontroller 328 configured to manage VM objects 307 and a pod VMlifecycle controller (PLC) 330 configured to manage pods 324. Acontroller 308 tracks objects in state database 303 of at least oneresource type. Controller(s) 308 are responsible for making the currentstate of supervisor cluster 101 come closer to the desired state asstored in state database 303. A controller 308 can carry out action(s)by itself, send messages to API server 302 to have side effects, and/orinteract with external systems.

Plugins 319 can include, for example, network plugin 312 and storageplugin 314. Plugins 319 provide a well-defined interface to replace aset of functionality of the Kubernetes control plane. Network plugin 312is responsible for configuration of SD network layer 175 to deploy andconfigure the cluster network. Network plugin 312 cooperates withvirtualization management server 116 and/or network manager 112 todeploy logical network services of the cluster network. Network plugin312 also monitors state database for custom objects 307, such as NIFobjects. Storage plugin 314 is responsible for providing a standardizedinterface for persistent storage lifecycle and management to satisfy theneeds of resources requiring persistent storage. Storage plugin 314cooperates with virtualization management server 116 and/or persistentstorage manager 110 to implement the appropriate persistent storagevolumes in shared storage 170.

Scheduler 304 watches state database 303 for newly created pods with noassigned node. A pod is an object supported by API server 302 that is agroup of one or more containers, with network and storage, and aspecification on how to execute. Scheduler 304 selects candidate nodesin supervisor cluster 101 for pods. Scheduler 304 cooperates withscheduler extender 306, which interfaces with virtualization managementserver 116. Scheduler extender 306 cooperates with virtualizationmanagement server 116 (e.g., such as with DRS) to select nodes fromcandidate sets of nodes and provide identities of hosts 120corresponding to the selected nodes. For each pod, scheduler 304 alsoconverts the pod specification to a pod VM specification, and schedulerextender 306 asks virtualization management server 116 to reserve a podVM on the selected host 120. Scheduler 304 updates pods in statedatabase 303 with host identifiers.

Kubernetes API 326, state database 303, scheduler 304, and Kubernetescontrollers 316 comprise standard components of a Kubernetes systemexecuting on supervisor cluster 101. Custom controllers 318, plugins319, and scheduler extender 306 comprise custom components oforchestration control plane 115 that integrate the Kubernetes systemwith host cluster 118 and VI control plane 113.

FIG. 4 is a block diagram depicting a logical view of a guest clusterexecuting in a virtualized computing system according to an embodiment.Supervisor cluster 101 is implemented by a software-defined data center(SDDC) 402. SDDC 402 includes virtualized computing system 100 shown inFIG. 1, including host cluster 118, virtualization management server116, network manager 112, shared storage 170, and SD network layer 175.SDDC 402 includes VI control plane 113 for managing a virtualizationlayer of host cluster 118, along with shared storage 170 and SD networklayer 175. A VI admin interacts with VM management server 116 (andoptionally network manager 112) of VI control plane 113 to configureSDDC 402 to implement supervisor cluster 101.

Supervisor cluster 101 includes orchestration control plane 115, whichincludes supervisor Kubernetes master(s) 104. The VI admin interactswith VM management server 116 to create supervisor namespaces includingsupervisor namespace 412. Each supervisor namespace includes a resourcepool and authorization constraints. The resource pool includes variousresource constraints on the supervisor namespace (e.g., reservation,limits, and share (RLS) constraints). Authorization constraints providefor which roles are permitted to perform which operations in thesupervisor namespace (e.g., allowing VI admin to create, manage access,allocate resources, view, and create objects; allowing DevOps to viewand create objects; etc.). A user interacts with supervisor Kubernetesmaster 104 to deploy applications on supervisor cluster 101 withinscopes of supervisor namespaces. In the example, the user deployscontainerized applications 428 on pod VMs 130 and non-containerizedapplications 429 on native VMs 140. Non-containerized applications 429execute on a guest operating system in a native VM 140 exclusive of anycontainer engine.

As described above, Kubernetes allows passing of configuration andsecret information to containerized applications 428. However, standardKubernetes does not extend this functionality beyond pod-based workloads(i.e., containerized applications executing in pods). Embodimentsdescribed herein extend this functionality for applications executing innative VMs (e.g., non-containerized applications 429). In embodiments,supervisor Kubernetes master 104 manages lifecycle of decoupledinformation 403 (e.g., config maps and secrets) for non-containerizedapplications 429. That is, supervisor Kubernetes master 104 performscreate, read, update, and delate operations on objects that includedecoupled information 403. Supervisor Kubernetes master 104 providesdecoupled information 403 to native VM controller 217 upon deployment ofnon-containerized applications 429 to native VMs 140. Native VMcontroller 217 cooperates with management agent 213 executing in eachnative VM 140 to provide decoupled information 403 for use bynon-containerized applications 429. Management agent 213 in each nativeVM 140 exposes decoupled information 403 for access by non-containerizedapplications 429. In embodiments, management agent 213 createsenvironment variables accessible by non-containerized applications 429.In embodiments, management agent 213 creates files in a filesystemaccessible by native VMs 140, which in turn can be read bynon-containerized applications 429. In some embodiments, the files canbe resident in system memory (e.g., RAM). Supervisor Kubernetes master104 can provide updates to decoupled information 403 to native VMcontroller 217, which in turn provides the updates to management agent213 for use by non-containerized applications 429.

When specifying a non-containerized application at supervisor Kubernetesmaster 104, the user can specify which decoupled information 403 uponwhich the application relies and how to consume the decoupledinformation (e.g., as environment variables, as files, etc.). SupervisorKubernetes master 104 schedules the non-containerized application to runin a VM object implemented by a native VM 140. Upon deployment of nativeVM 140, management agent 213 establishes a connect with native VMcontroller 217 using a hypervisor-guest channel (e.g., a virtual socketconnection). In embodiments, management agent 213 communicates withnative VM controller 217 over the hypervisor-guest channel using aremote procedure call (RPC) protocol. Management agent 213 sets updecoupled information 403 as specified for each non-containerizedapplication 429 (e.g., environment variables, files, etc.). Managementagent 213 updates decoupled information 403 exposed to non-containerizedapplications 429 as updates are received from supervisor Kubernetesmaster 104 through native VM controller 217.

FIG. 5 is a flow diagram depicting a method 500 of applicationorchestration in a virtualized computing system according to anembodiment. Method 500 can be performed by software in supervisorcluster 101 executing on CPU, memory, storage, and network resourcesmanaged by virtualization layer(s)(e.g., hypervisor(s)) or a hostoperating system(s). Method 500 can be understood with reference to FIG.4.

Method 500 begins at step 502, where supervisor Kubernetes master 104receives a specification for an application to be deployed using acustom VM object. At step 504, a user specifies configurationinformation and/or secret information for use by the application andspecifies how such information is to be consumed by the application.Supervisor Kubernetes master 104 creates a VM object for the applicationand config map/secret objects for the configuration/secret information.At step 506, supervisor Kubernetes master 104 deploys the application toa native VM 140 based on the specification data. In embodiments, theapplication is a non-containerized application that executes on a guestoperating system in native VM 140.

At step 508, management agent 213 receives config map/secrets fromsupervisor Kubernetes master 104 through native VM controller 217. Atstep 510, management agent 213 exposes the configuration/secretinformation in the config maps/secrets to the application as specifiedby the user. For example, at step 512, management agent 213 exposes theinformation as environment variables. In another example, at step 514,management agent 213 exposes the information as files (e.g., in afilesystem on storage or in memory). In still other examples, managementagent 213 performs a combination of steps 512 and 514. At step 516,management agent 213 updates the configuration/secret information asupdates are received from supervisor Kubernetes master 104 throughnative VM controller 217.

One or more embodiments of the invention also relate to a device or anapparatus for performing these operations. The apparatus may bespecially constructed for required purposes, or the apparatus may be ageneral-purpose computer selectively activated or configured by acomputer program stored in the computer. Various general-purposemachines may be used with computer programs written in accordance withthe teachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

The embodiments described herein may be practiced with other computersystem configurations including hand-held devices, microprocessorsystems, microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers, etc.

One or more embodiments of the present invention may be implemented asone or more computer programs or as one or more computer program modulesembodied in computer readable media. The term computer readable mediumrefers to any data storage device that can store data which canthereafter be input to a computer system. Computer readable media may bebased on any existing or subsequently developed technology that embodiescomputer programs in a manner that enables a computer to read theprograms. Examples of computer readable media are hard drives, NASsystems, read-only memory (ROM), RAM, compact disks (CDs), digitalversatile disks (DVDs), magnetic tapes, and other optical andnon-optical data storage devices. A computer readable medium can also bedistributed over a network-coupled computer system so that the computerreadable code is stored and executed in a distributed fashion.

Although one or more embodiments of the present invention have beendescribed in some detail for clarity of understanding, certain changesmay be made within the scope of the claims. Accordingly, the describedembodiments are to be considered as illustrative and not restrictive,and the scope of the claims is not to be limited to details given hereinbut may be modified within the scope and equivalents of the claims. Inthe claims, elements and/or steps do not imply any particular order ofoperation unless explicitly stated in the claims.

Virtualization systems in accordance with the various embodiments may beimplemented as hosted embodiments, non-hosted embodiments, or asembodiments that blur distinctions between the two. Furthermore, variousvirtualization operations may be wholly or partially implemented inhardware. For example, a hardware implementation may employ a look-uptable for modification of storage access requests to secure non-diskdata.

Many variations, additions, and improvements are possible, regardless ofthe degree of virtualization. The virtualization software can thereforeinclude components of a host, console, or guest OS that performvirtualization functions.

Plural instances may be provided for components, operations, orstructures described herein as a single instance. Boundaries betweencomponents, operations, and data stores are somewhat arbitrary, andparticular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of the invention. In general,structures and functionalities presented as separate components inexemplary configurations may be implemented as a combined structure orcomponent. Similarly, structures and functionalities presented as asingle component may be implemented as separate components. These andother variations, additions, and improvements may fall within the scopeof the appended claims.

What is claimed is:
 1. A virtualized computing system, comprising: ahost cluster having a virtualization layer directly executing onhardware platforms of hosts, the virtualization layer supportingexecution of virtual machines (VMs), the VMs including pod VMs andnative VMs, the pod VMs including container engines supporting executionof containers in the pod VMs, the native VMs including applicationsexecuting on guest operating systems; and an orchestration control planeintegrated with the virtualization layer, the orchestration controlplane including a master server and native VM controllers, the masterserver managing lifecycles of the pod VMs and the native VMs, the nativeVM controllers executing in the virtualization layer external to the VMsand configured as agents of the master server to manage the VMs, whereinthe native VMs include management agents executing therein to receivedecoupled information from the master server through the native VMcontrollers and to provide the decoupled information for consumption bythe applications executing in the native VMs, the decoupled informationincluding at least one of configuration information and secretinformation.
 2. The virtualized computing system of claim 1, wherein theapplications in the native VMs are non-containerized.
 3. The virtualizedcomputing system of claim 1, wherein the management agents are connectedto the native VM controllers through hypervisor-guest channels betweenthe native VMs and the virtualization layer.
 4. The virtualizedcomputing system of claim 3, wherein the management agents communicatewith the native VM controllers over the hypervisor-guest channels usinga remote procedure call (RPC) protocol.
 5. The virtualized computingsystem of claim 1, wherein the management agents expose at least aportion of the decoupled information to the applications via environmentvariables.
 6. The virtualized computing system of claim 1, wherein themanagement agents expose at least a portion of the decoupled informationto the applications via files stored in a filesystem accessible by thenative VMs.
 7. The virtualized computing system of claim 1, wherein themanagement agents are configured to receive updates to the decoupledinformation from the master server through the native VM controllers. 8.A host computer in a host cluster of a virtualized computing system, thehost comprising: a hardware platform; a virtualization layer, directlyexecuting on the hardware platform, supporting execution of virtualmachines (VMs), the VMs including pod VMs and native VMs, the pod VMsincluding container engines supporting execution of containers in thepod VMs, the native VMs including applications executing on guestoperating systems; and a native VM controller, executing in thevirtualization layer external to the VMs, configured as an agent of anorchestration control plane of the virtualized computing system, thenative VM controller configured to manage the native VMs, wherein thenative VMs include management agents executing therein to receivedecoupled information from the native VM controllers and to provide thedecoupled information for consumption by the applications executing inthe native VMs, the decoupled information including at least one ofconfiguration information and secret information managed by theorchestration control plane.
 9. The host computer of claim 8, whereinthe applications in the native VMs are non-containerized.
 10. The hostcomputer of claim 8, wherein the management agents are connected to thenative VM controller through a hypervisor-guest channel between thenative VMs and the virtualization layer.
 11. The host computer of claim10, wherein the management agents communicate with the native VMcontroller over the hypervisor-guest channel using a remote procedurecall (RPC) protocol.
 12. The host computer of claim 8, wherein themanagement agents expose at least a portion of the decoupled informationto the applications via environment variables.
 13. The host computer ofclaim 8, wherein the management agents expose at least a portion of thedecoupled information to the applications via files stored in afilesystem accessible by the native VMs.
 14. The host computer of claim8, wherein the management agents are configured to receive updates tothe decoupled information from the orchestration control plane throughthe native VM controllers.
 15. A method of application orchestration ina virtualized computing system including a host cluster having avirtualization layer directly executing on hardware platforms of hosts,the virtualization layer supporting execution of virtual machines (VMs),the VMs including pod VMs and a native VM executing on thevirtualization layer, the pod VMs including container engines supportingexecution of containers in the pod VMs, the native VM including anapplication executing on a guest operating system, the virtualizationlayer integrated with an orchestration control plane, the methodcomprising: receiving decoupled information at a management agent from amaster server of the orchestration control plane through a native VMcontroller, the management agent executing in the native VM and as anagent of the native VM controller, the native VM controller executing inthe virtualization layer; and providing the decoupled information forconsumption by the application executing in the native VM, the decoupledinformation including at least one of configuration information andsecret information.
 16. The method of claim 15, wherein the applicationin the native VM is non-containerized.
 17. The method of claim 15,wherein the management agent is connected to the native VM controllerthrough a hypervisor-guest channel between the native VM and thevirtualization layer.
 18. The method of claim 15, wherein the managementagent exposes at least a portion of the decoupled information to theapplication via environment variables.
 19. The method of claim 15,wherein the management agent exposes at least a portion of the decoupledinformation to the application via files stored in a filesystemaccessible by the native VM.
 20. The method of claim 15, furthercomprising: receiving updates to the decoupled information at themanagement agent from the master server through the native VMcontroller.